Papers with text-guided masking strategy
Semantics-enhanced Cross-modal Masked Image Modeling for Vision-Language Pre-training (2024.lrec-main)
Copied to clipboard
Haowei Liu, Yaya Shi, Haiyang Xu, Chunfeng Yuan, Qinghao Ye, Chenliang Li, Ming Yan, Ji Zhang, Fei Huang, Bing Li, Weiming Hu
| Challenge: | Existing methods for vision-language pre-training lack high-level semantics and text is not sufficiently involved in masked modeling. |
| Approach: | They propose a semantics-enhanced cross-modal MIM framework for vision-language representation learning that harvests high-level semantics from global image features via self-supervised agreement learning and transfers them to local patch encodings by sharing the encode space. |
| Outcome: | The proposed model achieves state-of-the-art or competitive performance on multiple vision-language tasks. |